Section: New Results

Deep-Temporal LSTM for Daily Living Action Recognition

Participants : Srijan Das, Michal Koperski, Francois Brémond, Gianpiero Francesca.

Keywords: Temporal sequences, Appearance, LSTM

We have proposed to improve the traditional use of RNNs by employing a many to many model for video classification. We analyzed the importance of modeling spatial layout and temporal encoding for daily living action recognition. Many RGB methods focus only on short term temporal information obtained from optical flow. Skeleton based methods on the other hand show that modeling long term skeleton evolution improves action recognition accuracy. In this work, we proposed a deep-temporal LSTM architecture (see fig. 16) which extends standard LSTM and allows better encoding of temporal information. In addition, we have proposed to fuse 3D skeleton geometry with deep static appearance. We validated our approach on publicly available datasets (CAD60, MSRDailyActivity3D and NTU-RGB+D), achieving competitive performance as compared to the state-of-the art. The proposed framework has been published in AVSS 2018 [39].

Figure 16. Framework of the deep-temporal LSTM proposed approach in  [39]